A storage system is a computer that provides storage service relating to the organization of information on writeable persistent storage devices, such as memories, tapes or disks. The storage system is commonly deployed within a storage area network (SAN) or a network attached storage (NAS) environment. When used within a NAS environment, the storage system may be embodied as a file server including an operating system that implements a file system to logically organize the information as a hierarchical structure of data containers, such as directories and files on, e.g. the disks. Each “on-disk” file may be implemented as a set of data structures, e.g., disk blocks, configured to store information, such as the actual data for the file. A directory, on the other hand, may be implemented as a specially formatted file in which information about other files and directories are stored.
The storage system may be further configured to operate according to a client/server model of information delivery to thereby allow many client systems (clients) to access shared resources, such as files, stored on the storage system. Sharing of files is a hallmark of a NAS system, which is enabled because of semantic level of access to files and file systems. Storage of information on a NAS system is typically deployed over a computer network comprising a geographically distributed collection of interconnected communication links, such as Ethernet, that allow clients to remotely access the information (files) on the storage system. The clients typically communicate with the storage system by exchanging discrete frames or packets of data according to pre-defined protocols, such as the Transmission Control Protocol/Internet Protocol (TCP/IP).
In the client/server model, the client may comprise an application executing on a computer that “connects” to the storage system over a computer network, such as a point-to-point link, shared local area network, wide area network or virtual private network implemented over a public network, such as the Internet. NAS systems generally utilize file-based access protocols; therefore, each client may request the services of the storage system by issuing file system protocol messages (in the form of packets) to the storage system over the network. By supporting a plurality of file system protocols, such as the conventional Common Internet File System (CIFS), the Network File System (NFS) and the Direct Access File System (DAFS) protocols, the utility of the filer may be enhanced for networking clients.
A SAN is a high-speed network that enables establishment of direct connections between a storage system and its storage devices. The SAN may thus be viewed as an extension to a storage bus and, as such, an operating system of the storage system enables access to stored information using block-based access protocols over the “extended bus”. In this context, the extended bus is typically embodied as Fibre Channel (FC) or Ethernet media adapted to operate with block access protocols, such as Small Computer Systems Interface (SCSI) protocol encapsulation over FC (FCP) or TCP/IP/Ethernet (iSCSI). A SAN arrangement or deployment allows decoupling of storage from the storage system, such as an application server, and some level of storage sharing at the application server level. There are, however, environments wherein a SAN is dedicated to a single server. When used within a SAN environment, the storage system may be embodied as a storage system that manages data access to a set of disks using one or more block-based protocols, such as SCSI embedded in Fibre Channel (FCP). One example of a SAN arrangement is described in U.S. patent application Ser. No. 10/215,917, entitled MULTI-PROTOCOL STORAGE APPLIANCE THAT PROVIDES INTEGRATED SUPPORT FOR FILE AND BLOCK ACCESS PROTOCOLS, by Brian Pawlowski, et al.
It is advantageous for the services and data provided by a storage system to be available for access to the greatest degree possible. Accordingly, some storage system environments provide a plurality of storage systems in a cluster, with a property that when a first storage system fails, the second storage system (“partner”) is available to take over and provide the services and the data otherwise provided by the first storage system. When the first storage system fails, the second partner storage system in the cluster assumes the tasks of processing and handling any data access requests normally processed by the first storage system. One such example of a storage system cluster configuration is described in U.S. patent application Ser. No. 10/421,297, entitled SYSTEM AND METHOD FOR TRANSPORT-LEVEL FAILOVER OF FCP DEVICES IN A CLUSTER, by Arthur F. Lent, et al. An administrator may desire to take a storage system offline for a variety of reasons including, for example, to upgrade hardware, etc. In such situations, it may be advantageous to perform a user-initiated takeover operation, as opposed to a failover operation. After the takeover operation is complete, the storage system's data will be serviced by its partner until a giveback operation is performed.
In a typical storage system cluster configuration, each storage system within the cluster contains a nonvolatile log (NVLog) that is stored in nonvolatile random access memory (NVRAM). Each storage system stores (“logs”) received data access operations directed to the storage system in its NVLog. In addition, each storage system forwards received data access operations to its partner storage system for storage in the partner's NVLog. Thus, during a failover operation, the surviving storage system may flush (write) the data associated with operations stored in the NVLog to storage devices, such as disks, associated with the failed storage system to bring those disks to a consistent state before initiating the processing of newly received data access operations, thereby enabling transparent failover operations while providing data consistency guarantees.
One scenario that may cause noted disadvantages in known cluster environments arises when a first storage system suffers an error condition that results in it entering a degraded mode. For example, a disk may fail, thereby resulting in the storage system initiating a RAID parity recomputation to recover the data contained on the failed disk. A subsequent error condition of the first storage system results in a failover operation of its data containers to a second partner storage system, which then suffers an error condition within a relatively short time period. However, the second storage system may have serviced data access requests directed to the data containers, with the appropriate logging of data in its NVLog. The rapid error condition of the second storage system causes that system to perform a giveback operation of the data containers to the first storage system prior to the containers being cleanly “shut down” by the second storage system i.e., before parity recomputation to recover any lost data. As noted, the NVLog data, including parity information, only exists on the second storage system; thus, when the first storage system re-initializes, it marks the data containers as dirty. If all of the storage devices associated with the data containers are present, the first storage system may perform appropriate parity computations to render the data containers as “clean”. However, if one or more storage devices are missing and/or inoperative, the first storage system must perform a complete file system check, resulting in considerable downtime and data unavailability, which may be on the order of hours and/or days depending on the size of the file system.